Automated Evaluation Methods with Attention to Individual Differences- a Study of a Computer-based Course in C

1 Tammy Rosenthal, Stanford University, Education Program for Gifted Youth, Ventura Hall, Stanford, CA 94305-4115 [email protected] 2 Patrick Suppes, Stanford University, Education Program for Gifted Youth, Ventura Hall, Stanford, CA 94305-4115 [email protected] 3 Nava Ben-Zvi, Hebrew University; Hadassah College Jerusalem, 37 Haneviim St., Jerusalem, Israel, 91010 [email protected] Abstract –– This study focuses on students’ performance in a distance-learning computer-based C course offered by EPGY (Education Program for Gifted Youth) at Stanford University. Its main goal is testing the feasibility of automated evaluation. Three automated methods consisting of a criterion set and weighted scoring methods were developed empirically, then applied to four hundred programs of ten types of problems and thoroughly analyzed. For most of the assignments, the more significant criteria for evaluation were related to execution time features, (i.e. number of operations and calls to functions executed by the program), memory storage required to store the variables of the program, and code’s length. The results showed that the criterion measures often differed by several orders of magnitude between students’ programs and were dependent on the specific problem domain. A low correlation was found between the evaluation criteria indicating that the criteria are nearly independent. Consequently, in qualitative tests for achievement and ability students’ performances exhibited high dimensionality.